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ABSTRACT 


This research establishes metrics for determining overall 





Navy secondary inventory forecasting accuracy when compared 





to actual demands at the Naval Inventory Control Point 











(NAVICP) . Specifically, two performance metrics are 
introduced: th averag performance index (API) and the 
median absolute deviation performance index (MPI). API 








measures forecasting accuracy of secondary inventory when 


compared against demand or forecast performance over a four 





quarter period. MPI measures the quarterly variability of 








forecast errors over the same period. 








The API and MPI metrics allow for the identification of 











poorly forecasted NAVICP secondary inventory items. The 





metrics can be applied to entire inventories or subsets of 








items based on type, demand, or cost. In addition, the API 





metric can be used to show overall inventory performance, 





providing NAVICP with a graphical means to assess 
forecasting performance improvements (or degradations) over 


time. 


The new forecasting accuracy methods developed in this 
research will allow the Navy to continually gauge the 


overall health of their inventory management practices and 





provide a method for improving forecasting accuracy. 











Additionally, they will assist NAVICP in complying with DoD 


directives that require NAVICP to monitor and continually 





develop improvements to inventory management practices. 
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EXECUTIVE SUMMARY 


This research establishes new metrics for quantifying 
Navy secondary inventory forecasting accuracy when compared 


to actual demands. Specifically, two performance metrics 








are introduced: the average performance index (API) and the 








median absolute deviation (MAD) performance index (MPI). 





API measures forecasting accuracy of secondary inventory 
when compared against demand performance over a one-year 


period. MPI is a measure of the variability of forecast 





errors over the same period. 











Once calculated, the API and MPI metrics are utilized 





in th development of an overall forecasting accuracy 









































method. Together, these methods allow for the 
identification of National Item Identification Numbers 
(NIINS) with poorly performing forecasts as well as the 
assessment of the overall performance of ali NAVICP 
secondary inventory items. Additionally this method 





provides NAVICP with a graphical means of illustrating 





overall forecasting performance from quarter to quarter that 





shows trends in forecasting performance over time. 


The new forecasting accuracy methods developed in this 
research will allow the Navy to continually gauge the 
overall health of inventory management practices and provide 


a method for improving forecasting accuracy. Additionally, 











the methods will assist NAVICP in complying with DoD 


directives which require it to monitor and continually 








develop improvements to inventory management practices (DoD, 
2004). Finally, improvements in forecasting accuracy should 
reduce excess inventory and thus save taxpayer dollars. 
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I. INTRODUCTION 


A. BACKGROUND 


The U.S. Navy performs inventory management on $18.7 
billion worth of secondary inventory. Secondary inventory 


consists of reparable components, consumable repair parts, 





bulk items and materiel, subsistence, and expendable end 





items, including clothing and other personal gear (DoD, 








2003). In an attempt to balance the needs for meeting 





warfighter readiness, coupled with competing resources and 





good stewardship, the Navy must actively manage these 





requirements on secondary inventory to effectively achieve 











inventory efficiency. The Department of Defense (DoD) 





Supply Chain Management Regulation requires all services to 


actively pursue efficient and effective materiel support at 





least cost (DoD, 2004). 








Currently, there is no mechanism in place to determine 
overall inventory efficiency for the U.S Navy. 
Specifically, the Navy has no established metrics to measure 
the accuracy or performance of its inventory management 


system. The Navy does not have a formal method to determine 





how well forecasted demand matches actual demand in the 
Navy’s secondary inventory. Because no methods or metrics 


have been established or utilized, the Navy has no way to 





determine overall inventory management ffectiveness. 





Therefore, the Navy is unable to track if improvements are 





being made to forecast accuracy or if a deficiency persists. 


B. GOVERNMENT ACCOUNTABILITY OFFICE (GAO) 





In 2007, the United States Government Accountability 





Office (GAO) conducted an audit of the Navy inventory system 





to determine its inventory management effectiveness. Among 
its findings, the GAO determined that the Navy held excess 
levels of inventory. The GAO identified $7.5 billon 





exceeded current requirements of the $18.7 billion in Navy 





secondary inventory for the years 2004 to 2007. About half 


of the $7.5 billion was identified as potential excess. The 





GAO also reported the Navy had not established the cost 





efficiency of its inventory management system over the 4 


year period, that the Navy’s demand forecasting 





ffectiveness is limited as requirements for items may 
change frequently after purchase decisions are made, and the 
Navy failed to adjust management practices in response to 


addressing unpredictability in demand (GAO, 2008). 





Although the Navy established metrics to measure 








warfighter support, it lacked metrics and goals for tracking 
and measuring the cost efficiency of its inventory 


management. Specifically, the Navy has not analytically 





evaluated th ffectiveness of its demand forecasting. 


Problems with demand forecasting include inaccurate and 





incomplete data, lack of communication between customers, 





item managers, and procurement personnel, and a failure to 
adjust management practices associated with initial 
procurement, on-order management, and stock retention (GAO, 


2008). 


The GAO recommended the Navy establish inventory 


performance metrics in order to track and identify areas for 





improvement and gaug th overall health of the Navy’s 
2 


inventory management practices. The GAO specifically 
recommended the Navy look to establishing a metric for 


measuring forecast and demand accuracy on their secondary 





inventory as this plays a vital role in determining the 








fiscal requirements on part inventory levels (GAO, 2008). 





Establishing such metrics would enable the Navy to track and 
improve demand forecasting processes and better accommodate 
fluctuations in demand. These improvements would enable the 


Navy to become mor fficient and also help quickly identify 








any deficiencies in its inventory management practices. The 











DoD concurred with the findings by the GAO and has begun 





exploring potential inventory concepts aimed to strengthen 


overall inventory management. 


Cc. THESIS OBJECTIVES 


This thesis seeks to establish a method to determine 


overall Navy inventory forecasting accuracy when compared to 





actual inventory demands. The creation of a forecasting 


accuracy method will enable the Navy to find a reference 





point for establishing inventory efficiency metrics and 


serve as the baseline for future improvements to inventory 





forecasting methods, thereby improving inventory management 








practices. This new forecasting accuracy method will allow 
the Navy to continually gauge the overall health of its 


inventory management practices. Critical outcomes are 





compliance with DoD regulations, improved effectiveness and 








responsiveness of the inventory system, and saving taxpayer 
dollars by reducing or eliminating excess inventory. An 


additional benefit of an improved inventory system would be 





to allow the Navy to reduc xpenditures on spare part 
stockpiles and make those funds available to other immediate 
3 


needs. Also, improved forecasting accuracy would reduce 








warehouse requirements and inventory storage costs. 


D. PAST THESIS WORK 
1. Repair Turn-Around Time 
Previous thesis studies conducted on inventory 


forecasting accuracy include a statistical analysis of the 





accuracy of the repair turn-around time (RTAT) forecast 





model at the Naval Inventory Control Point (Ropiak, 2000). 
This work sought to validate the Navy’s RTAT forecast model 


by comparing them to standard time series forecasting 





methods such as four-quarter moving average and exponential 


smoothing. 


The study determined that assumptions implicit in the 











Uniform Inventory Control Program (UICP) RTAT forecast model 


had a significant impact on forecast accuracy. It also 








indicated that the Navy’s model was no more accurate than 





the alternative standard techniques. 


This research, although closely related, is not a 
comparative study or validation of an established model. 
Rather, this thesis seeks to identify a baseline method for 
forecast accuracy metrics of the Navy’s entire secondary 


inventory. 


2. Shorebased Consolidated Allowance List 


A second thesis in this area of study focused on the 


development of an alternative demand forecasting model for 





Naval Aviation intermediat level inventories associated 











with the Shorebased Consolidated Allowance List, Yokosuka, 


Japan (Onders, 1994). The model consisted of two parts. 





The first consists of a causal model for forecasting demand 
originating from aircraft carriers utilizing flying hours 
and carrier deployment as independent variables. The second 
used a time series and marginal value method to forecast 


causal residuals and non-carrier demand. These two parts 





were then combined to produce a final forecast for an 


individual item. 





Onders (1994) differs from this study in that a 





comparative effort between models is not conducted in this 
thesis. Additionally, this study looks at both maritime and 


aviation secondary inventories and does not incorporate the 





use of causal modeling techniques. 


3% Repair Turn-Around Time Modeling 


Lastly, past thesis work was conducted examining the 


forecast accuracy for a subset of Navy inventory repair 








parts data utilizing a computed RTAT by the UICP (Santos, 





2003). This research consisted of running bootstrap 
Simulations with various repair part arrival rates to 
compare forecasting accuracy. Santos (2003) differs from 
this study as repair part inventory items are not the only 
items analyzed in this thesis for determining forecast 
accuracy. Additionally, this study looks at the entire 
population of secondary inventory vice an analysis on a 


subset of inventory items as was done in Santos (2003). 
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II. DATA 


A. SOURCES OF DATA 


This thesis utilizes historical data from two Uniform 








Inventory Control Program (UICP) systems managed at the 





Naval Inventory Control Point Philadelphia (NAVICP-P) for 








aviation inventories and the Naval Inventory Control Point 








Mechanicsburg (NAVICP-M) for maritime inventories. The UICP 





system was developed in the 1960s to assist in materiel 








controls at the Naval Inventory Control Points (ICPs). The 
data includes both consumable and repairable inventory 


items. 


Consumable parts consist of those items that are 





discarded after utilization or at the end of their usable 








service life. Repairable parts consist of both aviation and 
maritime parts that undergo a reutilization process where 


items are refurbished and put back into servic when 








advantageous for financial or materiel availability 





purposes. Normally, repairable items are returned to 


serviceable condition after forwarding to designated 





overhaul facilities at significantly less cost and in much 





shorter lead times rather than going through a full 


procurement process. 





The UICP program provides automated data processing in 
support the following NAVICP functions (NAVSUP Pub 542, 
2001): 





a. Provisioning 





b. Database Maintenance 


oe Stock lists and cataloging 

















Ot Configuration management 
es Allowance 

£. Load lists 

g. Requisition processing 

‘abe Transaction reporting 

nee Inventory review 

Re Repair and program management 
k. Stratification/utilization 
dy Purchase 

m. Financial control 

n. Accounting and disbursing 
O. Simulation and research 

jo Materiel readiness 

Gi. Data Retrieval 














The UICP supports the NAVICP functions of inventory 


review, stratification/utilization, and data retrieval for 





this research effort. 


B. UICP 








The UICP system has since undergone numerous software 
updates and hardware upgrades since its inception and is 
currently in the process of being replaced by the Navy 


Enterprise Resource Planning system (Navy ERP). The Navy 





ERP Single Supply Solution, which consolidates the wholesale 


and retail supply functions of the Navy, was deployed in 


March of 2010 to Naval Supply Systems Command (NAVSUP) .1 





Although the UICP is a legacy system that is being 





phased out of NAVICP daily inventory management functions, 


the data remains viable for this analysis since the data 








contained in the UCIP systems are being utilized to populate 





the Navy ERP databases. 


Cc. DATA FIELDS 


For the purposes of this thesis, data was extracted 


from both the NAVICP-P and NAVICP-M UICP databases. Each 











set of historical data includes quarterly demand and 





forecast histories by individual National Item 




















Identification Numbers (NIIN) and includes the following 





additional data fields: 




















FIELD DESCRIPTION 
National Item Identification Number - a 
NIIN unique 9-digit numeric code identifying an 





item in the NATO Codification System. 























Cognizance Code - a two digit alphanumeric 
code used to identify the responsible 
COG inventory manager, the stores account and 
the type of materiel. A list of COG codes 
are found in NAVSUP P-485, Vol , Appendix 
18. 
PLT Procurement Lead-Tim overall time an item 


takes to be procured. 











Production Lead-Tim period between the 
PRLT placement of an order and receipt of the 
item into the supply system in quarters. 














RTAT Repair Turn-Around Time - time (in quarters) 











1 Navy ERP, http://www.erp.navy.mil/deployment_info.html 
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FIELD DESCRIPTION 





an item spends in the repair system. 














Process Repair Turn-Around Time —- the amount 
PRTAT of time an item takes to be repaired in 
quarters. 
Replacement 
Full cost to procure an item (in USD). 
Prauce 





Repair Price 


Price for a repairable item to be repaired 

















(in USD). 

Demand Total wmumber of demande for an -Lthem, in. a 
quarter. 

Forecast Forecasted number of total demands for an 





item in a quarter. 





D 





Table 1. Description of UICP data fields. 





DATA USED IN THIS RESEARCH 





The aviation data from NAVICP-P is comprised of the 


fields in Table 1 for forty-one consecutive quarters of 














demand history by individual NIINs, covering the period 





December 2000 to February 2010. Additionally, the aviation 


data also includes twenty-one consecutive quarters of 











forecasting history by individual NIINs (March 2005 to March 








2010). The maritime data from NAVICP-M has the same fields 


in Table 1 for forty consecutive quarters of demand history 





from June 1999 to May 2009. The maritime data also includes 








twenty-two consecutive quarters of forecasting history by 





individual NIIN from March 2005 to June 2010. 











E. 


NAV 


DATA SUMMARY 


The following is a summary of the contents of both the 


ICP-M and NAVICP-P data from the UICP systems: 
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Type # NIINs # COGs 








Aviation 41,005 9 
Maritime 208,824 35 





Table 2. Data description. 


































(COG) 


Aviation data is broken down by Cognizance Code 
in Figure 1. Maritime data is presented in Figure 2. 
Aviation parts by COG type 

@1R 
m7R 
=0Q 
m9B 
mOR 
m6R 

m= Other 

Figure 1. Distribution of aviation NIINs by COG code. 
Maritime parts by COG 
3670 
on m1H 7H 
13500 
| a7G m 6X 
m7E m2S 
\ yy m Other 
—————— 
Figure 2. Distribution of maritime NIINs by COG code. 
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The COG code is a two-position code to identify and 





designate the inventory control point, which exercises 
supply management control. The first position of the 


cognizance symbol is numeric and identifies the stores 





account. The second position is a single-letter code that 





designates the inventory manager or ICP that has control of 








the materiel. The most commonly used cognizance symbols for 





the Navy are: 


1R (Aviation) and 1H (Maritime) - Consumable parts 





7R (Aviation) and 7H (Maritime) - Repair parts. 





A complete list of COG code types and their respectiv 





inventory managers can be found in Appendix 18 of the NAVSUP 








P-485, Naval Supply Procedures, Volume 














Maritime parts have a significantly larger set of COG 





codes compared to aviation parts. This is primarily due to 





the broad number of ship classes as well as the number of 


unique characteristics found within each ship class (e.g., 

















DDG Flt I, DDG Flt , DDG Flt A, etc.) as compared to 















































aircraft. Maritime data taken from the NAVICP-M UICP system 





consisted of 208,824 NIINs from 35 different COG codes. 











Table 3 presents summary statistics from the relevant 





fields of both aviation and maritime UICP systems. 


Avg Avg Avg Avg Avg Avg Avg Avg 
Type Qtrly Qtrly Qtrly OQOtrly Repl Repair Qtrly Qtrly 
PLT PRLT RTAT PRTAT Price Price Demand Forecast 





Aviation 3.49 4.64 0.36 0.42 $10,426 $5,163 3.34 4.15 
Maritime 3.419 4.46 0.82 0.86 $16,284 $10,627 1.50 i Eesrons) 


Table 3. Summary statistics for aviation and maritime 
secondary inventory. 
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III. INVENTORY FORECAST METRICS 


This chapter develops metrics useful for determining 


Navy inventory forecast accuracy. The goal is to provide 














NAVICP with metrics useful for identifying NIINS whose 





forecasts differ significantly from actual demand. Studies 





in the field of inventory management have focused on the 





application of time series models for measuring forecasting 





accuracy. Commonly used metrics include the mean absolute 
percentag rror (MAPE) and the mean square error (MSE) 
(e.g., Tersine, 1998). However, this research does not use 


these metrics for a couple of reasons, including that they 





assess the fit of a single time series model to a single 








series, while NAVICP is most interested in evaluating 














forecasts in one time period over many NIINs. In addition, 











MAPE is not practical at NAVICP as it causes calculation 





errors due to division by zero. Therefore, an alternative 


approach is used in this research study to measure forecast 





CEEOE's 


A. DEVELOPING PERFORMANCE INDICATORS 


The simplest performance metric is just the difference 





between the forecast and demands over some time period. Let 














Fi,; denote the forecast for NIIN i for period j and let Di,; 




















denote the demand for the same NIIN and period, which could 











be a quarter or year. Then the performance index (PI) for 








NIIN i in period j is the difference between forecast and 











demand, expressed mathematically as: 


PI, , =F, —D,,;- (1) 
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The problem with this type of simple performance index 


is that it does not provide a context within which to judge 





the difference. For example, the PI is the same for one 














NIIN that had a forecast of 10,000 items and a demand of 





10,010 and another NIIN that has a forecast of 1 item anda 














demand of ll. That is, in both cases PI = -10, meaning the 
forecast was low by 10 items. However, in the first case 
the forecasting algorithm performed well, satisfying 10,000 


out of 10,010 demands, or 99.9%, while in the second case 





the forecasting algorithm performed poorly, only fulfilling 





1 out of 11 demands, or 9%. 


This suggests Equation (1) should be modified to 





compare the difference between forecast and demand for time 
period j to demands for the same period, relative to the 


demand: 
= (2) 


The performance index can now b interpreted as the 




















fraction of demand either over-forecast (if PI is positive) 
or under-forecast (if PI is negative). However, while the 
Equation (2) performance index works in the previous 
example, it has two difficulties. First, when demands are 0 
it is unclear what the PI value should be. Second, the 
calculated outputs are not symmetric in changes to the 














forecast versus changes to the demand. That is, for a fixed 


level of demand 


while 
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On the other hand, for a fixed forecast 
7 7D); 
PI,,=—_—+>-l as D,j,>0 
while 


FSD»: 
PI,,=—1_*>? as D,,-0. 


ij 





The problem with the performance index being undefined 
when demands are zero is easily fixed by adding one to the 
denominator, 


Fs eae 


— (3) 
sa D, +1 


? 


but the asymmetry in performance remains and, in fact, is 





now more evident. That is, for a fixed level of demand 
F,,-D,, D, 
PI,, =—1_*>~-— as F,,-70 
pe 2 ted Dal 
while 
F, —D, j 
PI,,=— “>0 as Fj 
7 D,,+1 


On the other hand, for a fixed forecast 


Fj-D, 
PI_, =—__+ > -1 as D,, © 


nd D,,+1 
UJ 
while 
i —D, j 
PI, = = , ee as D,,—0 
+ 
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In words, the above expressions say that with this 





formulation of the performance index, if the forecast is 
"far off" in the sense that demand is much larger than the 
forecast, the worst the index can be is -l. On the other 
hand, in the reverse case where forecast greatly exceeds 


demand, then the index can get arbitrarily large. This has 











the effect of making one type of error (forecast larger than 





demand) look much worse than the other type of error (demand 


larger than forecast). 


Now, one might propose to change the denominator from 


demands to the forecast, as in: 


Fi 7? 
Pla i (4) 
i 


but this does not eliminate the asymmetry, it simply flips 








it into the other direction so that the error when demand is 
larger than forecast looks much worse than the error when 


forecast is larger than demand. 





A solution to this dilemma is to use both Equations (3) 


and (4) depending on whether the forecast is greater than 





demand or vice versa. Specifically, when F,,2D,, calculate 


the performance index as: 


F_,-D,, 
PI,, = (5) 
D,, +1 

UJ 


and when F,,<D,,then calculate the performance index as: 


F..-D,, 
PI a J 1,J ( 6) 
ve es a 
I,J 


Equations (5) and (6) can be combined into one equation as 
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PI, , =i 1 x {F, 2 D,,}+—2 —- x {F< D,,}, (7) 


where I {o} is the indicator function; it equals 1 when the 


condition in the brackets is true and it eguals 0 otherwise. 


The performance index in Equation (7) can be 





interpreted as the fraction of excess inventory compared to 





demand (when PI is positive) or as the fraction of unmet 








demand compared to the forecast (when PI is negative). Note 
how it behaves. For a fixed level of demand, Dj;,,>0, 
F,,-D,, F,,-D,, 
PI, , = 24x 1 F, , 2 D, ,} + —* x {F< D,;} 30 as Fj; 90 
2 D+ ; F,,+1 


while for a fixed forecast, F;,;>0, 


PI, =H > D, ;}+t x, <D,,}>-«0 as D,, >. 


In words, the above expressions say that, with this 








formulation of the performance index, if either the forecast 


or demand is far off then the index can get arbitrarily 





large in either the positive or negative direction. 








To show the performance index in Equation (7) provides 


a symmetric measure of performance, whether the forecast is 





greater than demand or vice versa, is perhaps best 
illustrated through example. Consider the following 


forecasts and demands in Table 4. 








Qtr1 Qtr 2 
Forecast 20 9 
Demand 9 20 











Table 4. Hypothetical forecast and demand data by quarter. 
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Table 5 shows the performance indices using the 





asymmetric methods of Equations (5) and (6) versus the 


symmetric method of Equation (7). This table shows that 





whether the forecast is greater than demand by 9 units, as 





shown in Qtr 1, or if forecast is less than demands by 9 


units, as in Qtr 2, Equation (7) produces results in an 





index of the same magnitude where sign indicates whether 


























demands were over-forecasted (+) or under-forecasted (-). 
Method Qtr 1 performance Qtr 2 performance 
Equation (5) Pl 1.10 -0.52 
Equation (6) PI 0.52 -1.10 
Equation(7) Pl 1.10 -1.10 
Table 5. Performance indices calculated on Table 4 using 


hypothetical data. 





A second, mor xtrem xample illustrates the 





asymmetry ffect ven more. That is, using the more 
divergent demand and forecast data in Table 6, Table 7 shows 


the performance indices using Equations (5) through (7). 








Qtr1 Qtr 2 
Forecast 120 5 
Demand 5 120 








Table 6. Hypothetical forecast and demand data by quarter. 

















Method Qtr 1 performance Qtr 2 performance 
Equation (5) Pl 19.16 -0.95 
Equation (6) PI 0.95 -19.16 
Equation (7) Pl 19.16 -19.16 
Table 7. Performance indices for Table 6 data. 
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Notably, the performance indices from Equations (5) and (6) 
again demonstrate their asymmetry whereas Equation (7) shows 


continued symmetry. 





The asymmetry arising in Equations (5) and (6) and the 
symmetry of Equation (7) are even more evident when plotting 
actual demand and forecast data. For example, using demand 


and forecast data from the March through May 2006 NAVICP-P 





aviation data, Figure 3 plots the performance indices from 


Equations (5) and (6) versus demand for the same period. 








Demand 
40 60 80 
1 


Demand 


20 
1 














Performance Index Performance Index 


Figure 3. Performance indices versus demand for Mar-May 2006 
NAVICP Aviation data for Equation (5) on the left 
and Equation (6) on the right.? 





Note how in the left plot the performance indices are 
bounded below by -1 and in the right plot they are bounded 
above by +1. 


2 Note: On the right side plot, white space in the lower left region 
occurs due to the fact that PI in Equation (5) cannot be less than -D,, 
which occurs when F;=0. (the lowest possible value of F;). 
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In comparison, when applying Equation (7) 


data, Figure 4 shows the 





to the same 


symmetry of this performance 
metric. 





Demand 














Performance Index 


Figure 4. Performance indices for May-Mar 2006 NAVI] 











[CP 
Aviation data using Equation (7) .3 
In a previous study conducted by NAVICP (Bencomo, 
2010), a formula for measuring forecasting error (referred 





to as the "Lead-time Adjusted Symmetric Error: LASE") 


Similar to Equation (7) was proposed: 


LASE = Fi 





3 Note: 
the lower 
cannot be 


As in the right side plot of Figure 3, the white space in 


left region occurs due to the fact that PI in Equation (7) 
less than —-D; which occurs when F,=0. 
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Although Equation (8) is a symmetric method for 
measuring performance, it also contains a "smoothing effect" 
as forecasting accuracy degrades. For example, using the 
hypothetical forecast and demand data from Table 8, Table 9 


compares the two performance indices. 














Qtr1 Qtr 2 
Forecast 20 100 
Demand 5 5 





Table 8. Hypothetical set of quarterly forecast and demand. 

















Comparison PI Qtr 1 PI Qtr 2 
Equation (8) PI 1.11 1.78 
Equation (7) PI 2.50 15.83 
Table 9. Comparative PI’s on Table 8 sample data using 





Equation (7) and Equation (8). 


By averaging forecast and demand in the denominator of 





Equation (8), NAVICP’s metric produces a larger smoothing 


effect compared to the Equation (7) performance index. 





Essentially, Equation (8) underestimates the magnitude of 





how far off the forecast is from the demand as it includes 


both the forecast and demand terms in the denominator of the 





metric. 





This effect is illustrated in Table 9, where the NAVICP 
Equation (8) performance index increases from 1.11 to 1.78 
while the Equation (7) performance index increases from 2.50 
to 15.83. Notice, the forecast and demand were more 
divergent from quarter 1 to quarter 2 as the performance 


metric increase was larger in the Equation (7) calculation 








compared to NAVICP’s LASEK calculation in Equation (8). The 
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smoothing effect when using NAVICP’s LASE method makes 




















identification of any large divergence between forecast and 
demand difficult compared to utilization of the PI 
calculation. 

Note how Equation (8) behaves in extreme cases. For a 


fixed level of demand, D;,;>0, 








'.-D.. 
= a »2 as F,>0 


(F,, + D,,,) a4 I,J 
z 


LASE = 





while for a fixed forecast, F;,;>0, 








7 Pi, 


Fst Pid) 
2 





LASE = >—2 as D,,>™. 








In both cases, where demand was either massively over- 


forecast or under-forecast, the NAVICP performance index 


converges to either plus two or minus two. The result is 








that the NAVICP metric does not have an intuitive 


interpretation and it makes increasingly poor forecast 


performance hard to distinguish. 


Consider the following example in which demand is 











ten, 


D;,j=10. If F;,j=10 then the NAVICP performance index is 0 


indicating the forecast matched the demand perfect 


tly. 





However, if the forecast is double the demand (Fy, j=20), 





the NAVICP performance indicator is 0.625, while if 


then 


the 


forecast. 28: 10 tames- -demand. (rs 7=100). “then. the “NAVICP 





performance indicator is 1.6. Finally, if the forecast is 





1,000 times demand (F;,j;=10,000) then the NAVICP performance 





indicator is 1.995. Thus when the forecast got five times 





worse, going from F;,j;=20 to F;,j;=100 the index increased by 


22 





just under one, yet when the forecast got five hundred times 
worse, going. from. -Fy,7=20 to: Fy,=10,000: the index. increased 


by just over 1.3. 





Also note that the NAVICP performanc index does not 





distinguish between an under- and over-forecast. That is, 








the NAVICP performance index is the same whether D,,,;=10 and 


F;,4=100 or D;,j=100 and Fy;,j;=10. However, there are different 





consequences, and thus presumably different inventory 
management strategies and options, depending on whether an 


item is over- or under-stocked. 





In summary, Equation (7) is the preferred performance 














index form as it better identifies those NIINS whose 











forecast and demands are significantly divergent. Further 





details of NAVICP’s analysis can be found in Bencomo (2010). 


B. QUANTIFYING AVERAGE PERFORMANCE: 
THE ANNUAL PERFORMANCE INDEX (API) METRIC 














Measuring NIIN forecast performance on a quarterly 


basis is likely to be misleading as the forecasts are 








calculated to meet an average demand. Thus, the first 











metric, referred to as the Annual Performance Index (API), 


is based on four quarters of forecast and demand data. The 











API metric follows the same tenets used in the quarterly PI 








of Equation (7) which ensures the API is a symmetric measure 





regardless of whether forecast is greater than demand or if 
demand is greater than forecast. The main difference is 


that demands and forecasts are summed over four quarters: 











The choice of four quarters to sum over is based on the 





assumption that some NIINs might have annual cycles in their 











demand patterns and thus using a multiple of four quarters 








would minimiz th ffects of such a cycle. It is also 





motivated by the idea that, while it is desirable to smooth 





over some reasonable length of time in order to mitigate the 





effects of temporary spikes or lulls in demand, using too 





long of a period would have th ffect of incorporating 
historical demand patterns and/or forecasts that may not be 
relevant to current inventory performance. Whether four 
quarters is the best choice is an open question and not 


studied in this thesis. 


Equation (9) is based on the idea of comparing the 








total demands over the past four quarters to the total 





forecast over the same period. This seems like a sensible 








approach from the perspective that in each quarter there ar 
a specific number of demands and each of those quarterly 
demands correspond to a particular quarterly forecast. 
However, an alternate approach would be to compare the 


forecast from a specific quarter to all of the demands that 








occurred in some future period of time. Implementing this 





requires a small modification to Equation (9). For example, 
if in quarter j one wanted to assess the performance of the 
forecast from a year ago in quarter j-3, one would just 
multiply that forecast times four and compare it to the 


demands over that same period: 
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a (10) 
j-3 
4x F, i 3 > Ds, j-3 
“I ¥ TS4xF <> D., 
4x F,+1 oS ee 
i,j J 





This thesis will use the API from Equation (9), though 








NAVICP could use the multiplicative forecast approach form 





of Equation (10) if preferred. 


Cs QUANTIFYING PERFORMANCE VARIATION: THE MEDIAN ABSOLUTE 
DEVIATION PERFORMANCE INDEX (MPI) METRIC 














Once an individual NIIN’s quarterly PI is computed 

















using Equation (7), it may be of interest to identify NIINs 




















whose forecasts regularly diverge from demand. The median 
absolute deviation (MAD) is a measure of statistical 
dispersion. It is preferred to other measures of variation, 
such as the standard deviation, as it is more resilient to 
outliers. For k observations, the MAD is calculated as 

MAD = median(|X, - X|,|X, - X|,-...|X, -X]), (11) 

















where X =median(X,,....X,)- 





For example, to compute the MAD using Equation (11), 





consider the following data set: 


> The median of the data set is 2. 


> Taking the absolute deviations about 2 
produces the data set {2, 1, 0, 1, 6}. 
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e Sorting the newly computed absolute deviations 
from low to high results in the following data 


set: 





From this, the median is 1 and thus the MAD of the original 


data set is l. 





The MAD can be used to quantify the variation in 


quarterly performance indices in order to identify those 




















NIINS that are regularly and routinely off in their 














forecasts versus those NIINs that are only intermittently 








off. Presumably, the forecasts in the former case are 


likely to be easier to correct than those in the latter 








case. Perhaps more importantly, the MAD will help identify 




















those NIINs with forecasts that are on average correct, say 
on an annual basis, but that are routinely off on a 


quarterly basis, say due to churn. For purposes of this 





thesis, churn is identified as two quarters of over and 





under forecasting in a given four-quarter period. A high 














MAD value indicates a wide dispersion between a NIIN’s 








quarterly forecasts and demands. This helps in identifying 





NIINS with more volatility in their forecast accuracy and a 














NIIN’s potential for the occurrence of churn. 








The use of MAD over other statistical methods, such as 


the standard deviation, is preferred as it is less 





influenced by occasional spikes or one-time anomalies in 
forecasting performance. For example, consider the 


following hypothetical quarterly performance indices for 











some NIIN: 





The standard deviation of the above performance indices 
is 149.3 while the MAD is 0.5. The one large performance 


index of 300 results in a much larger standard deviation 





compared to MAD. If 300 were due to a one-time demand 














spike, one would not want to identify the NIIN as having 





excess quarterly forecast variation. Rather, the measure 


should be insensitive to a single deviation but sensitive to 





two or more deviations, which is what the MAD does for a 


year's worth of guarterly performance indices. 
To illustrate this, consider the following dataset: 
{1, 300, 1, 200} 


For this data, the standard deviation remains almost the 





same at 149.4 while the MAD significantly increases to 99.5. 











The standard deviation minimally changes and does not 








Significantly differentiate between the two datasets while 
the MAD does. 








Applying a MAD calculation to a set of quarterly PIs 


























provides a NIIN’s’ forecast variability, which will be 
referred to as the MAD performance index or MPI. Given that 
the average procurement lead time for a NIIN is 











approximately 4.5 quarters (Bencomo, 2010), and because som 














NIINs likely experience an annual cycle in demand, Equation 











(12) shows how MPI will be calculated on a year's worth of 








quarterly performance index data.4 Thus, MPI is defined as 


MPI, = MAD(PI, ,, PI, 


i,j-1? 


PL cagPlox), (12) 


ij? i,j-2? 


4 Note that as the number of quarters used in the MAD calculation is 
increased, the MAD requires a greater number of quarters with large 
performance indices before it indicates excess quarterly variation. For 
example, if the MAD is based on six quarters of performance index data, 
two or more out of the six will have to be large before the MAD is 
affected. 
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Additionally, 


set of quarterly demands by N 


allows a 


demand levels 





quarter 


D. 


Two metrics hav 


differentiation 


SUMMARY 





over 


to quarter. 


those with 


b 

















N provides a method 








N with consis 





between Ns 








intermittent demands 














n developed in this chapter: 





applying a similar MAD calculation to a 





that 











Cent 


from 


API 





as a 


d by 


and MPI. The API measures the average forecast performance 
over some period of time. Herein the period is defined 
year (four quarters) but that can be modified as desire 
NAVICP. The MPI measures the quarterly deviation 





forecast from demand and it 





those N 











according 


Ns 


to 


the 











quarter-by-quarter. 


Adoption of these metrics will 


to flag N 














further examination and correction. 








Ns that exceed certain API 


API, 





should provide insight 


that may be performing well on average, 


but have significant deviat 





give NAVICP the abi 








and MAD thresholds 





Examination thresh 

















could be set based on staff workload capabilities, 
cost, or COG code. Such an approach gives NAVICP 
capability to focus attention on those NIINS 
demonstrate poor forecasting performance, enab 


improvements to the overall inventory management system. 
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IV. APPLICATION OF THE METRICS 





This chapter illustrates the application of the API and 








MPI to inventory forecast evaluation. In the first section, 














hypothetical NIIN demand and forecast data are compared 











using the API and MPI metrics to provide some insight into 


how the metrics work and what they indicate about forecast 





performance. The second section then demonstrates how to 


apply the metrics to a complete inventory to identify actual 














NIINsS that have problematic forecasts. 








A. ILLUSTRATING HOW THE METRICS WORK 


To illustrate how the metrics work, consider the four 











hypothetical NIINs in Table 10, and their demand and 








forecast history from June 2005 to May 2006. These are 











NIINS with forecasts that ar "well behaved" in the sense 








that the forecasts closely match the demands quarter by 


quarter. 


Jun - May Forecast and Demand quantities by quarter 


Jun-Aug Sept-Nov Dec-Feb Mar-May 


NIIN Forecast Demand | Forecast Demand | Forecast Demand | Forecast Demand 
012345678 6 
123456789 0 
234567890 8 
345678901 0 




















Table 10. Quarterly forecast and demand for four 
hypothetical NIINs. 

















Using Equation (7), Table 11 gives the API metrics for 





the four NIINs. 
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NIIN API 
012345678 -0.13 
123456789 -1.00 
234567890 0.05 
345678901 0.00 
Table 11. API metrics for NIINs in Table 10. 

Her Ww s relatively small API indices, which we 
interpret as forecasts and demands closely matching from 
quarter to quarter. The largest magnitude difference is 
NIIN 123456789, where the -1.0 indicates that the NIIN was 














100% under-forecast over the four quarters 


zero for all 

















(the forecast was 


four quarters while there was a demand of one). 








units, 


basis the forecast for this N 


than the previous N 


because 


In contrast, 


but that was out of 25 demands, 


N 











N 012345678 actually under-forecast by three 





























SO on a percentage 


N actually performed better 

















N. Of course, one might argue that 
of the low demand NIIN 123456789 is of little 
but as will be shown later in this section, such 





interest, 





N 











Using Equation (12), Table 12 gives the MPI 
the four NIINs. 
NIIN MPI 
012345678 0 
123456789 0 
234567890 1 
345678901 0 
Table 12. MPI metrics for NIINs 
As MPI quantifies a N 
quarter to quarter. The small MPI values 























Ns can be identified by plotting API 





versus demand. 


metrics for 












































in Table 10. 








N’s forecast variability from 





in Table 12 
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indicate that demand and forecast performance numbers remain 


relatively consistent quarter to guarter. This is validated 











in Table 10 where the NIIN forecast and demand numbers 








remain relatively close to each other in at least two or 











more quarters for each NIIN. 








Now, consider another example in Table 13, where the 











hypothetical NIINs have very poor forecasts showing churn. 


Jun - May Forecast and Demand quantities by quarter 


Jun-Aug Sept-Nov Dec-Feb Mar-May 





NIIN Forecast Demand | Forecast Demand | Forecast Demand | Forecast Demand 
912345678 1500 4 6 5000 4000 1 
923456789 15 3500 3300 1500 900 
934567890 500 
945678901 














Table 13. Quarterly forecast and demand for four 
hypothetical NIINs with churn. 


























It becomes vident upon examination of the computed 


APIs and MPIs in Table 14 and Table 15 that churn exists 














because API is low but MPI is very high. 



























































NIIN API 
912345678 0.00 
923456789 0.04 
934567890 0.03 
945678901 -0.04 
Table 14. API Metrics for the NIINs in Table 13. 
NIIN MPI 
912345678 506.31 
923456789 95.89 
934567890 83.00 
945678901 269.50 
Table 15. MPI Metrics for the NIINsS in Table 13. 
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Most notably, the significantly large MPI numbers are 




















due to th xcessively large variation and significant 
swings in over- and under-forecasting performance from 
quarter to quarter for each NIIN in Table 13. However, when 


the demand and forecast numbers are aggregated to calculate 




















their respective API’s, the NIINs have very small API 








indices because their four quarter forecast and demand sums 


are very close. 





In yet another example, Table 16 shows a set of 











hypothetical NIINs whose quarterly forecast and demand data 














reflect two patterns. The first two NIINs in Table 16 show 








intermittent demand or forecast spikes while the last two 














NIINsS show consistently high under- and over-forecasting for 








at least two or more quarters. The resulting API and MPI 





calculations are provided in Tables 17 and 18. 





Jun - May Forecast and Demand quantities by quarter 


















































Jun-Aug Sept-Nov Dec-Feb Mar-May 
NIIN Forecast Demand | Forecast Demand | Forecast Demand | Forecast Demand 
812345678 50 50 8000 2 4 4 
823456789 5000 15 16 
834567890 5000 1 0 
845678901 0 
Table 16. Quarterly forecast and demand for hypothetical 
NIINsS with spikes in demand or forecast and large 
variability. 
NIIN API 
812345678 98.75 
823456789 -70.44 
834567890 -1999.70 
845678901 7600.00 
Table 17. API metrics for NIINs in Table 16. 
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NIIN MPI 
812345678 0.02 
823456789 0.10 
834567890 500.50 
845678901 200.00 
Table 18. MPI metrics for NIINs in Table 16. 

















As shown in Tables 17 and 18, intermittent spikes in 


quarterly demand or forecasting quantities produce a large 











API but low MPI indices. The low MPI is due to the MAD, 











which is less sensitive to the occasional outlier such as a 





one-time spike in demand. However, if forecasting or demand 


quantities are significantly off for two or more quarters in 











a four-quarter period, as demonstrated by the last two NIINs 

















in Tables 17 and 18, they produce both high API and MPI 


indices. 


B. APPLYING THE METRICS TO ACTUAL DATA 








Plotting the MPI and API indices for NIINs allows for a 























graphical depiction of NIIN- performance. Figure i) 








demonstrates such a graph for NAVICP-P data from the period 








June 2005 to May 2006 where MPI is plotted on the y-axis and 








API is plotted on the x-axis). 


33 


Churn region 


Significant under 
forecasting region 





Significant over 
forecasting region 
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Figure 5. Plot of API vs. MPI for NAVICP-P NIINs for the 
period June 2005 —- May 2006. 

In Figure 5, the highlighted regions that depict areas 
where NIINsS are significantly under- or over-forecast. In 
addition, the churn region is identified in Figure 5 where, 
for example, NIIN 010490325 is identified. Seven additional 





N 








Ns, 





over-forecasting region, 





are also ide 


three in the under-forecasting region and four in the 


ntified. Table 19 then 





shows the actual forecast and demand data where it is clear 





that N 











N 010490325 experiences churn 


with three quarters of 


Significant over-forecasting and one quarter of significant 


under-fo 


recasting. 
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June 2005 — May 2006 Forecast and Demand quantities by quarter? 


Jun-Aug 


NIIN 


Forecast 


Demand 


Forecast 


Sept-Nov 


Demand 


Forecast 


Dec-Feb 


Demand 


Mar-May 


Forecast Demand 





014776024 
014780782 
014769165 


467 
1876 
449 


0 


010490325 
014919544 
014919564 
014434294 


649 
1441 
11 


0 


1056 


1850 


1773 
952 
710 
396 


0 
175 0 
014432801 0 



































Table 19. Quarterly Forecast and Demand quantities 
for NIINs listed in Figure 5. 
Additionally, notice the NIINs identified in the 











highlighted under-forecast region in Figure 5, and compare 





them against their corresponding forecast and demand 











Here we see those N 





quantities in Table 19. Ns residing in 


the under-forecast region are indeed 





xperiencing 


significant under-forecasting as demonstrated by the 


consistently high demand numbers in each quarter with zero 





look at wN 








forecast quantities. Conversely, if we N’s 








residing in th over-forecasting region as identified in 








in this case 








Figure 5, you see the opposite effect, as NIINs 








contain significantly large forecasting quantities against 


zero demands. 


This methodology can also b 





selectively applied by the 

















subsetting to particular NIINs of interest, say by COG code 








with NIINS over a certain 








or perhaps by price (for example, 





cost threshold). Figure 6 shows the resulting plots for 











NIINS separated by consumable and repairables for June 2005 














through May 2006 aviation data. Notably, repairable NIINs 





-) Forecasting numbers rounded to nearest whole number. 
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located on the right side plot of Figure 6 exhibit a much 














tighter grouping over the consumable NIINs in the left plot 
suggesting that the repairables as a group have better 


forecasts than the consumables. 








1000 
1000 


100 
* 
100 


Log (MPI) 


Log (MPI) 


10 
10 














10,000 -10,000 -100 0 100 10,000 





Log (API) Log (API) 














Figure 6. Comparison between consumable aviation NIINs on 
left against repairable aviation NIINS on right for 
the period June 2005 -—- May 2006. 




















Similarly, Figure 7 shows a comparison of NIINs that 




















cost less than $10,000 versus NIINs that cost $10,000 or 


more. 
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Figure 7. Aviation NIINs that cost less than $10K on left 
compared against aviation NIINs that cost $10K or 
more on right for the period June 2005 - May 2006. 



































Note that for the NIINsS costing $10K or more _ show 
better performance than those costing less than $10K in the 


sense that the left side plot in Figure 7 is larger and more 








dispersed when compared to the right side plot in Figure 7. 








This suggests that NAVICP-P appropriately places more 


scrutiny on forecasting accuracy on the mor expensive 








NIINS. 











That said, the right side plot is still useful for 











identifying expensive NIINs that are not performing as well 

















as equivalent NIINsS by simply tightening the thresholds and 

















flagging NIINS whose MPI and API performance indices are 




















outside the norm for expensive NIINs. Identifying those 

















NIINsS with poor forecasting performance in this fashion, so 





they can be audited by NAVICP personnel, should facilitate 


incremental forecasting performance improvements. 
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The API metric can also be used to summarize overall 





inventory performance. This would allow NAVICP to capture 
overall forecasting performance for all secondary 


inventories and determine if forecasting performance is 





improving or degrading over time. It is accomplished by 





graphically displaying the API percentiles by quarter over 











time, wher th median, 75th percentile, and 95th 





percentiles for the API metrics for the over- and under- 











forecasted NIINsS are plotted by quarter. 











Figure 8 illustrates the idea using the NAVICP-P 
aviation data from 2006 to 2008. The top plot is for all 














the NIINs with over-forecasts and improving forecasts across 





th ntire inventory would be visible as a downward trend in 














the bars over time. The bottom plot is for all the NIINs 





with under-forecasts and improving forecasts across the 











ntire inventory would be visible as an upward trend in the 


bars over time. 


The top of the grey bar indicates the 95th percentile 





of API for the particular quarter. This means that 95 














percent of all the aviation NIIN APIs were less than or 





equal to that value. Similarly, the yellow and dark gray 





bars show the 75th and 50th percentiles, respectively. 








Therefore, in the first quarter of 2006, 95% of all over- 





forecast aviation NIINs had APIs less than 3.58, 75% had 














APIs less than 0.68, and 50% had APIs less than 0.08. Thus, 























50% of th over-forecast NIINs have excellent API metrics 








(less than 0.08). On the other hand, 5% of the NIINsS have 

















APIs greater than 3.6, which means those NIINs' have 








forecasts that are more than 3.5 times the number of 


demands. 
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Using Figure 8, NAVICP can determine whether 





forecasting performance is improving or degrading over time 


as depicted by the trend line in both charts. A trend line 





that tends towards zero indicates improvement in forecasting 





performance over time. This method allows NAVICP to 
generate a graphical illustration for showing overall 


forecasting performance of secondary inventory. 


Cc. SUMMARY 








Application of the API and MPI metrics and their 





utility has been show in this section. Incorporating the 





use of the API and MPI metrics provides NAVICP with a method 








to measure forecasting performance. This method allows for 














the identification of NIINs that exceed certain established 











thresholds for further examination and correction. 
Examination thresholds could be based on, but not limited 


to, staff workload capabilities, item cost, or COG codes. 





Lastly, a method using the API indices is demonstrated and 








allows NAVICP the capability to capture overall quarterly 


forecasting trends based on the statistical spread of the 











quarterly API. This provides NAVICP with insight into 


whether forecasting performance is improving or degrading 





over time. 
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V. CONCLUSIONS 


This research establishes metrics for determining 


overall Navy secondary inventory forecasting accuracy when 





compared to actual demands at the Naval Inventory Control 





Point (NAVICP). Specifically, two performance metrics were 














introduced: the API and the MPI. API measures forecasting 


accuracy of secondary inventory when compared against demand 





or forecast performance over a four-quarter period. MPI 


measures the quarterly variability of forecast errors over 





the same period. 


The API and MPI metrics allow for the identification of 














poorly forecast NAVICP secondary inventory items. The 


metrics can be applied to entire inventories or subsets of 











items based on type, demand, or cost. In addition, the API 





metric can be used to show overall inventory performance, 





providing NAVICP with a graphical means to assess 
forecasting performance improvements (or degradations) over 


time. 


The new forecasting accuracy methods developed in this 
research will allow the Navy to continually gauge the 
overall health of their inventory management practices and 


provide a method for improving forecasting accuracy. 











Additionally, they will assist NAVICP in complying with DoD 


directives that require NAVICP to monitor and continually 











develop improvements to inventory management practices (DoD, 


2004). 
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